The Policy Improvement Algorithm : General Theory

The average cost optimal control problem is addressed for Markov decision processes with unbounded cost. It is found that the policy improvement algorithm generates a sequence of policies which are c-regular (a strong stability condition), where c is the cost function under consideration. This result only requires the existence of an initial c-regular policy, and an irreducibility condition on the state space. Furthermore, under these conditions the sequence of relative value functions generated by the algorithm are bounded from below, and \nearly" decreasing, from which it follows that the algorithm is always convergent. Under further conditions, it is shown that the algorithm does compute a solution to the optimality equations, and hence an optimal average cost policy. These results shed new light on the optimal scheduling problem for multiclass queueing networks. Surprisingly, it is found that the formulation of optimal policies for a network is closely linked to the optimal control of its associated uid model. Moreover, the relative value function for the network control problem is closely related to the value function for the uid network. These results are surprising since randomness plays such an important role in network performance.